Identifying the potential miRNA biomarkers based on multi-view networks and reinforcement learning for diseases

Abstract MicroRNAs (miRNAs) play important roles in the occurrence and development of diseases. However, it is still challenging to identify the effective miRNA biomarkers for improving the disease diagnosis and prognosis. In this study, we proposed the miRNA data analysis method based on multi-view miRNA networks and reinforcement learning, miRMarker, to define the potential miRNA disease biomarkers. miRMarker constructs the cooperative regulation network and functional similarity network based on the expression data and known miRNA–disease relations, respectively. The cooperative regulation of miRNAs was evaluated by measuring the changes of relative expression. Natural language processing was introduced for calculating the miRNA functional similarity. Then, miRMarker integrates the multi-view miRNA networks and defines the informative miRNA modules through a reinforcement learning strategy. We compared miRMarker with eight efficient data analysis methods on nine transcriptomics datasets to show its superiority in disease sample discrimination. The comparison results suggested that miRMarker outperformed other data analysis methods in receiver operating characteristic analysis. Furthermore, the defined miRNA modules of miRMarker on colorectal cancer data not only show the excellent performance of cancer sample discrimination but also play significant roles in the cancer-related pathway disturbances. The experimental results indicate that miRMarker can build the robust miRNA interaction network by integrating the multi-view networks. Besides, exploring the miRNA interaction network using reinforcement learning favors defining the important miRNA modules. In summary, miRMarker can be a hopeful tool in biomarker identification for human diseases.


INTRODUCTION
MiRNAs are small endogenous non-coding RNAs regulating the gene expression at the posttranscriptional level [1].Studies have shown that miRNAs play crucial roles in many important biological processes, including cell differentiation and proliferation [2], tumorigenesis and metastasis [3].It is helpful to explore the close associations between miRNAs and human diseases for understanding the disease pathogenesis.Moreover, miRNAs show many advantages as the potential disease biomarkers, such as excellent tissue specificity, high reliability of condition identification and the favorable stability in body f luids [4].Hence, identifying the miRNA biomarkers related to the disease occurrence and development is of great significance for disease diagnosis and prognosis.
Biological network analysis provides the strong support for understanding the changes of biomolecular interactions during the disease development as well as defining the potential biomarkers [5].Common network analysis methods are based on the molecular correlation.Zhang et al. [6] proposed the weighted gene co-expression network analysis (WGCNA) to explore the coexpression patterns of genes based on the weighted correlation network.It defines the dense modules by a weighted topological overlap measure.WGCNA has been adopted in the disease studies for other different omics, such as transcriptomics [7][8][9].The dynamic network biomarker (DNB) method evaluates the dynamic changes of molecular correlation and the SD of the molecules to define the potential disease biomarkers [10,11].The DNB can highlight the critical tipping points in the disease progression.These network-based analysis methods provide reference for miRNA network construction based on expression data.
In addition to the miRNA expression data, various miRNArelated knowledge bases strongly support the understanding of the miRNA functional interactions [12].Those miRNAs having many common regulated target genes or diseases may function similarly.Some methods evaluate the miRNA interactions by considering the experimentally validated miRNA-target relations [13][14][15].The intuitive way for evaluating the miRNA functional interactions is calculating the proportion of the common targets by Jaccard index [16].However, it discards a lot of similarity information.Some methods try to adopt the Gene Ontology (GO) annotations or protein-protein interactions (PPIs) as the additional information of target genes for inferring the miRNA interactions [17][18][19].Unfortunately, the GO database has the limited annotation information for massive target genes, adverse to evaluate the miRNA interactions accurately [16].Meanwhile, there are lots of false positives without the experimental validity in PPI databases, which may introduce the bias for calculating the miRNA functional similarity.The miRNAdisease relations [20][21][22] provide the direct evidence of miRNAs participating in the occurrence and development of diseases, becoming the strong choice for evaluating the miRNA functional interactions.
One miRNA usually regulates the development processes of multiple diseases, and one disease is associated with many miR-NAs.The key of disease-based miRNA functional similarity calculation is assessing the disease semantic similarity accurately.Wang et al. [23] proposed a graph-based method to infer the disease semantic similarity based on the disease directed acyclic graphs (DAGs) from the public database Medical Subject Headings (MeSH, https://www.ncbi.nlm.nih.gov/mesh).The diseases had high semantic similarity if they had large common parts in their DAGs.Nevertheless, the DAG-based disease similarity ignores the specific disease semantic significance by equally evaluating the importance of diseases with the same distance to root disease [24].Therefore, to obtain the miRNA functional interactions with high quality, the disease semantic information in the miRNA-disease relations should be rationally assessed and fully used.
What's more, it is conducive to construct the robust miRNA interaction network by using the miRNA expression data and the information in knowledge bases simultaneously.The miRNA expression data contain the change information of miRNA cooperative regulation between different sample groups.And the known miRNA-disease relations sustain the assessment of miRNA functional similarity strongly.Thus, integrating the miRNA cooperative regulation and miRNA functional similarity can promote the downstream network analysis tasks, including the potential biomarker identification for diseases.
Defining the informative subnetworks based on molecular interaction networks is an important topic in biological network analysis [25].Some methods adopt the clustering techniques to divide the biological networks [6,11].In general, a predefined cluster number is needed, but it is difficult to determine.Some methods adopt the heuristic strategy for identifying the important modules.Zhang et al. [26] proposed the network-based game theory method (NGTM) to identify the potential cancer subnetwork biomarkers by evaluating the feature contribution using the cooperative game theoretic metrics (Shapley values) in the heuristic-based module extension.But in the heuristicbased module identification methods, the heuristic information used is rather limited, making it easy to obtain local optimal results.Different from supervised and unsupervised learning, reinforcement learning (RL) aims to make decisions maximizing the long-term returns [27].Accordingly, the RL strategy may bring more possibilities for module biomarker identification by fully exploring the solution space to obtain global optimal results.Paim et al. [28] tried to employ RL to detect the communities in complex networks and proposed the Q-Learning [29] for community detection (QLCD) method.Each node in the network acts as an agent, which selects one node from its nearest neighbor nodes (action space) to compose the clusters.The agent nodes learn the action policy to maximize the network modularity.However, QLCD may fail to find the competitive modules in disease studies due to its inherent simple action space and inadequate learning strategy.It is necessary to further explore the potential of reinforcement learning in defining the network biomarkers for diseases.
For identifying the potential miRNA disease biomarkers effectively, we proposed the miRNA data analysis method based on multi-view networks and reinforcement learning, miRMarker.It constructs the miRNA cooperative regulation network based on expression data.The miRNA functional similarity network is built using the known miRNA-disease relations from public knowledge bases.Then, miRMarker integrates the two miRNA networks and defines the crucial miRNA modules by a reinforcement learning strategy.We verified the effectiveness of miRMarker in disease sample discrimination by comparing it with eight effective data analysis methods on nine transcriptomics datasets.In addition, we examined the potential miRNA module biomarkers defined by miRMarker for colorectal cancer.The experimental results demonstrate the great potentials of miRMarker in defining the important module biomarkers for the diagnosis and prognosis of diseases.

MATERIALS AND METHODS
miRMarker consists of two main parts: (i) constructing the miRNA networks based on miRNA expression data and miRNA-disease relations, respectively, integrating the two networks and (ii) defining the crucial miRNA modules by a reinforcement learning strategy.Figure 1 shows the workf low of miRMarker.

MiRNA expression datasets
In this study, nine miRNA expression datasets were collected to evaluate the effectiveness of miRMarker.All the datasets were derived from the public data repository Gene Expression Omnibus (GEO, https://www.ncbi.nlm.nih.gov/geo/),involving multiple diseases, such as hepatocellular carcinoma, thrombocytosis and colorectal cancer.Table 1 gives the details of the nine miRNA expression datasets.Datasets GSE41574, GSE67139, GSE32273, GSE34496, GSE41282 and GSE108153 are two-class datasets.Datasets GSE31164, GSE39046 and GSE35834 are multi-class datasets.We mapped all human mature miRNA names to the standard miRNA accession numbers using miRBase v22.0 [30] (see more details in Supplementary Materials available online at http://bib.oxfordjournals.org/).The expression values of the probes representing the same miRNA were averaged.

MiRNA-disease relations
The manually curated miRNA-disease relations were extracted from the two largest knowledge bases miRCancer [20] and miR2Disease [21].miRCancer uses the text mining technique to extract the miRNA-cancer associations from the medical literatures in the PubMed database (https://pubmed.ncbi.nlm.nih.gov/), and then the associations are revised manually.miR2Disease provides the comprehensive regulatory associations between miRNAs and human diseases curated from the published papers.We downloaded the latest versions of miRCancer (9080 entries, downloaded in June 2022) and miR2Disease (2877 entries, downloaded in July 2022).All human mature miRNA names were mapped to the standard miRNA accession numbers by miRBase

MiRNA network construction
miRMarker constructs the miRNA interaction network by integrating the cooperative regulation network G CR and the functional similarity network G FS .G CR is constructed using the miRNA expression data.G FS is built based on the miRNA-disease relations.
Let F = f 1 , f 2 , ..., f m represent the feature (miRNA) set and m be the number of features.miRMarker defines the combination feature f ij = f i − f j for two miRNAs f i and f j (1 ≤ i < j ≤ m), representing the relative expression relationship between f i and f j [31].The dysregulation of expression level of f ij from normal samples to disease samples describes the change of cooperative regulation of f i and f j .The Mann-Whitney U test was employed to measure the differential significance of f ij between the two sample groups (Kruskal-Wallis Test for multi-class samples).Then, the weight of the edge between f i and f j is defined as the negative logarithm value of the P-value of f ij (P − value ij ) in U test.If the difference of f ij is significant between the two different sample groups, the edge weight between f i and f j is large.This means the cooperative regulation relationship between f i and f j is strong in G CR .
the set of edge weights.miRMarker constructs the functional similarity network G FS based on miRNA-disease relations.Figure 2 shows the calculation process of miRNA functional similarity between f i and f j .A miRNA corresponds to a regulated disease set containing many human diseases.The miRNA functional similarity between f i and f j is defined as the similarity between the corresponding regulated disease sets is the v-th disease regulated by f j .p and q are the numbers of diseases in D i and D j , respectively.We employed the natural language processing (NLP) model BioBERT [32] to embed the disease terms into numeric vectors.BioBERT is a powerful NLP model trained on the massive biological  The calculation of miRNA functional similarity sim F f i , f j considers the similarity between each pair of the diseases in D i and D j .The functional similarity between f i and f j is high if the similarity between the disease sets D i and D j is high.
the set of edge weights.miRMarker defines the miRNA interaction network G FN by integrating the cooperative regulation network G CR and functional similarity network G FS .G CR and G FS are both undirected complete graphs.All the edge weights in G CR and G FS are normalized using min-max normalization, respectively.In G FN , the edge weight of f i , f j is the average value of normalized edge weights of f i , f j in G CR and G FS : where w CR f i , f j and w FS f i , f j are the normalized edge weights of f i , f j in G CR and G FS , respectively.G FN is a complete graph containing many uninformative edges with low weights.miRMarker defines a subnetwork of G FN by preserving the informative edges.The maximum spanning tree (MST) algorithm [33] where , f a indicates the Q value (accumulated reward) of selecting the action node f a when the state node is f s .A larger Q value means more accumulated rewards.miRMarker selects an action node from the action space, that is, the set of neighbor nodes of the visited nodes in S t .The epsilon-greedy strategy is adopted to handle the explorationexploit problem.When the current state node is f s , the action node is randomly selected from the action space with a probability ε.Otherwise, miRMarker selects the action node f a leading to the largest Q t f s , f a .ε is decayed with episodes as follows: where ε f is a small value, typically set as 1e−05 [28].e is the current episode number.K indicates the maximum number of episodes.miRMarker fully explores the action space in the early episodes, tending to select a random action node.ε decreases exponentially as e increases.miRMarker gradually reduces the selection of random action nodes but selects the optimal action node, that is, the node leading to the max Q value in action space.The selected action node f a is added to S t .
The reward value of adding f a to S t considers the classification performance and size of S t simultaneously as follows: where where α is the learning rate and γ is the discount rate.Q t f s , f a is updated greatly if α is large, meaning the information learned by interacting with the environment play a major role.If α is small, Q t f s , f a gets a few updates.max Q t f a , f a indicates the maximum Q value of selecting any action node f a when the state node is f a .The updating of Q t f s , f a makes use of the immediate reward, the discounted maximum expected accumulated reward and the current estimation of the Q value simultaneously.f s is updated using the latest action node f a .Then, the new action node is selected iteratively.The current episode stops when AUC (S t ) reaches the maximum value of 1.000 or |S t | exceeds a predefined limit.At the end of all the episodes, miRMarker has explored the miRNA network sufficiently and learned the valuable information for optimal action policy.miRMarker defines the essential miRNA modules using the learned Q-table.The module extension process starts from each initial state node f t .Initially, the module M t only contains f t .The optimal action node f a was selected iteratively using the epsilongreedy strategy based on the learned Q-table.The selected action node f a is added to M t .The state node is changed to f a .There are two conditions of stopping the module extension: (i) the AUC of module M t reaches the maximum value of 1.000, which means the performance of M t has no further improvement, and (ii) the new action node f a has been added to M t during the module extension.
No new node will be added in this situation.
Algorithm 1 shows the pseudocode of module identification in miRMarker.Lines 1-2 initialize the visited node set S t and Qtable Q t , respectively.Line 3 initializes an episode.Line 4 sets the ε at the current episode.Line 5 starts a search procedure.Line 6 defines the state node.Line 7 selects the action node by the epsilon-greedy strategy.Line 8 adds the action node to S t .Line 9 evaluates the classification performance of S t .Line 10 calculates the reward value.Line 11 updates the Q-table.Lines 12-14 judge the stop conditions of searching.Line 17 identifies the module based on the learned Q-table.
miRMarker identifies one module for each of the g initial state nodes.On each identified module, a support vector machine (SVM) classifier [34] with a linear kernel is built.The weighted voting is employed to integrate the prediction probability values of g classifiers.The weights of g classifiers are defined using the normalized average AUCs of modules in an inner 5-fold crossvalidation on training samples.Then, miRMarker labels new samples by the weighted voting of the g classifiers.

Involved statistics
Several statistics are used in the proposed method.In constructing the cooperative regulation network, the Mann-Whitney U test is employed to measure the differential significance of the combinatorial feature f ij between the two different sample groups (Kruskal-Wallis Test for multi-class samples).In the construction of a functional similarity network, the cosine similarity is adopted to calculate the disease semantic similarity between two disease embedding vectors.In module identification, the Mann-Whitney U test is adopted to calculate the individual distinguishing ability of features for defining the initial nodes (Kruskal-Wallis test for multi-class samples).Large differences of feature expression levels between the two groups indicate the high individual distinguishing ability of a feature.
SVM-RFE is an efficient feature selection method that integrates the SVM and recursive feature elimination to select the informative features.DNB was proposed for time series data analysis [11].In this study, we adopted DNB to analyze the static data.The hierarchical clustering was employed to obtain the potential feature groups firstly [11].Then, DNB constructed the Pearson correlation networks on each sample group, respectively, and calculated the critical index (CI) score for each feature group by evaluating the changes of molecule correlation and the SD between different correlation networks.The feature group with the maximum CI score is defined as the DNB biomarkers.GroupBN employs the hierarchical clustering technique to group the similar features and constructs the Bayesian network to model the relationships between the feature groups for probabilistic inference.DDRM evaluates the miRNA synergistic relationships by considering the co-regulated target genes and non-co-regulated targets simultaneously and then constructs the weighted miRNA synergistic network.It found 200 miRNA functional modules using the kernel clustering method.For a specific disease study, DDRM maps the corresponding miRNA expression data to the miRNA functional modules, evaluates each module based on the mapped miRNAs and defines the key modules for the disease.We adopted the top five functional modules with the maximum average performance in an inner 5-fold cross-validation on training samples as the DDRM biomarkers.NGTM was proposed for known gene regulatory networks [26].QLCD was designed for existing complex network datasets [28].For NGTM and QLCD, we constructed the correlation networks based on the miRNA datasets by symmetrical uncertainty (SU) [37].SU is a widely used entropy-based similarity measure.Each node was linked to its k neighbors with the highest SU values.The k value was searched in {1, 3, 5, 7, 9} to make the node degrees in SU network fit the power law distribution best.
In this study, the NLP technique was adopted to obtain the disease semantic vectors for calculating the miRNA functional similarity based on the known miRNA-disease relations.Thus, we also compared miRMarker with the method using the disease DAG-based miRNA functional similarity network (named as N-DAG) [23], aiming to show the effectiveness of miRNA functional similarity calculation in miRMarker.
The effectiveness of miRMarker was evaluated on binary classification and multi-class classification, respectively, for distinguishing the different sample groups.The receiver operating characteristic (ROC) analysis was conducted to evaluate the classification performance (macro-averaged one-versus-rest ROC for multi-class classification).The 10-fold cross-validation was performed 10 times to get the average area under the curve (AUC), sensitivity and specificity.
Dataset GSE32273 consists of the samples from three different tissues, thus, it was split into three subsets depending on tissue types.For binary classification, we also used the multi-class datasets (GSE31164, GSE39046 and GSE35834) by dividing them into multiple binary subsets.Consequently, there were a total of 17 binary datasets/subsets included in the binary classification.
In miRMarker, the k value was searched in {3, 5, 7, 9} to make the node degrees in G FN−MST−kNN fit the power law distribution best.The maximum episode number K in module identification was set as 2000.The learning rate α was set as 0.2.The discount rate γ was set as 0.8.The number of initial state nodes was set as 5.The size limit of the visited node set S t is set as 50.
For all the feature selection methods, the SVM with linear kernel was adopted as the classifier.Unit-variance (UV) scaling was performed to nondimensionalize the features before training the models.All methods were implemented in R 4.2.0.

Performance evaluation of miRMarker
We compared miRMarker with eight efficient data analysis methods SVM-RFE, WGCNA, DNB, GroupBN, NGTM, QLCD, N-DAG and DDRM in ROC analysis.Table 2 shows the comparison results in AUC in binary classification.Bold font marks the highest AUC values on the corresponding dataset.Mark ' * ' means the AUC value of the method is significantly different from that of miRMarker (t-test, P-value < 0.05).row 'Average' shows the average AUC values for each method over the 17 binary datasets.The row 'Win/Tie/Loss' lists the numbers of datasets on which the average performance of miRMarker is significantly higher than, not significantly different from, or significantly less than that of the comparison method.miRMarker performed better than other comparison methods on most cases.Compared with SVM-RFE, miRMarker outperformed it on 10 of the 17 datasets in AUC.In comparison with the network-based data analysis methods, miRMarker defeated WGCNA, DNB, GroupBN, NGTM, QLCD and DDRM on 10, 14, 12, 16, 13 and 8 datasets, respectively.miRMarker outperformed N-DAG on 7 datasets and tied with it on 10 datasets.It can be seen that miRMarker achieved the highest AUCs (bold fonts) on 9 of the 17 datasets.Besides, miRMarker obtained the highest average AUC (0.914) over the different datasets and the lowest average SD (0.110).Tables S1 and S2, available online at http:// bib.oxfordjournals.org/,show the comparison results in sensitivity and specificity, respectively.It can be seen that miRMarker obtained the highest values on eight and eight datasets, respectively.In addition, miRMarker obtained the highest average sensitivity (0.948) and specificity (0.922) over datasets.Additionally, we examined the comparison results between miRMarker and other data analysis methods in the Matthews correlation coefficient (MCC) (Table S3 available online at http://bib.oxfordjournals.org/).miRMarker got the highest average MCC (0.662) over datasets.The comparison results in binary classification illustrated the effectiveness of miRMarker.Table S4 available online at http://bib.oxfordjournals.org/gives the comparison results for multi-class classification.The comparison results in multi-class classification also indicated that miRMarker showed more advantageous performance than other methods.
miRMarker integrates the multi-view miRNA networks constructed on the expression data and the known miRNA-disease

Ablation study
miRMarker builds the miRNA interaction network by integrating the cooperative regulation network and the functional similarity network.We compared miRMarker with the methods only use the cooperative regulation network (named as N-CR) or functional similarity network (named as N-FS) for validating the effectiveness of network integration.Figure S1 available online at http:// bib.oxfordjournals.org/shows the comparison results in average AUC values over datasets.miRMarker obtained better performance than N-CD and N-FS.The integration of the cooperative regulation network and functional similarity network uses the multi-view miRNA interactions well to obtain the robust miRNA interactions.It is helpful for finding the critical information for disease sample discrimination.

Parameter sensitivity analysis
There are two main parameters in miRMarker: (i) the learning rate α and (ii) the number of initial state nodes g (module numbers).The setting of α concerns the update range of Q values in the module identification.Figure S2A available online at http://bib.oxfordjournals.org/shows the average AUC values of miRMarker for different settings of α in {0.2, 0.4, 0.6, 0.8}.It can be seen that the performance of miRMarker decreases slightly when α changes from 0.2 to 0.8, meaning miRMarker is relatively stable under the different settings of α.A small learning rate α may result in better performance, which is consistent with the previous study [28].α = 0.2 was adopted in this study.The setting of module number g inf luences the effects of ensemble strategy in disease sample discrimination.Figure S2B available online at http://bib.oxfordjournals.org/shows the average AUC values of miRMarker for different settings of g in {1, 3, 5, 7}.The performance of miRMarker shows great improvements continuously as g changes from 1 to 5.However, the performance change of miRMarker becomes f lat when g changes from 5 to 7, indicating the performance is hard to improve further.g = 5 is applicable to most cases and adopted in this study.

Biomarker identification on colorectal cancer dataset
We performed miRMarker to identify the potential miRNA biomarkers for colorectal cancer diagnosis on a real-world transcriptomics dataset from the GEO database (GSE108153).Colorectal cancer is a global malignant tumor facing many risk factors such as smoking, unhealthy diet and obesity [38].Current treatment methods for colorectal cancer, including radiotherapy and resection, have limited inf luences on the cure rate and prognosis.The improvement in diagnosis and prognosis of colorectal cancer is still challenging and attractive.
The discovery set (GSE108153) contains 21 pairs of cancer and adjacent normal tissues collected from the colorectal cancer patients.Three cancer transcriptomics datasets (GSE112264, GSE113486, GSE211692) involving multiple solid cancers were retrieved from the GEO database.For each of them, we extracted the colorectal cancer samples and non-cancer samples as the independent validation sets.Table S5 available online at miRMarker defined five miRNA modules on the discovery set (Figure 3A-E).Each of the identified modules is a star graph.They showed good performance in distinguishing the cancer samples from normal samples (Figure 3F).
Figure 4 shows the ROC curves of miRMarker on the discovery set and three independent validation sets.For distinguishing the colorectal cancer samples from normal samples, miRMarker obtained an AUC of 1.000 on the discovery set.On the three independent validation sets, the AUC values were 1.000, 0.9995 and 0.996, respectively.The defined modules performed outstandingly in cancer sample discrimination on the discovery set and independent validation sets, showing the great potential as the biomarkers for colorectal cancer.
Furthermore, we explored the relationships between the defined miRNA modules and colorectal cancer by constructing the module-disease network (Figure 5).All the five defined modules are connected to the disease node of colorectal neoplasms, ref lecting that the defined modules are closely related to the occurrence and development of colorectal neoplasms.More details about the module-disease network are given in Supplementary Materials available online at http://bib.oxfordjournals.org/.
Highly connected hub nodes usually locate at the key positions in biological networks, which play important roles in biological processes [39].The pathway analysis was conducted for the target genes of the five hub miRNAs (hsa-miR-135b-5p, hsa-miR-224-5p, hsa-miR-183-5p, hsa-miR-96-5p and hsa-miR-195-5p) using the online tool DAVID [40,41].Figure 6 shows the top 30 enriched pathways of Kyoto Encyclopedia of Genes and Genomes (KEGG) [42].Most of the top enriched pathways are associated with cellular growth and senescence, signal transduction, viral or bacterial infection and cancer-related metabolism.Typically, 51 target genes were enriched in the KEGG pathway 'Colorectal cancer' (hsa05210) with a false discovery rate (FDR) of 2.55e−06 (Table S6 available online at http://bib.oxfordjournals.org/),indicating the significant roles of the regulated target genes in the occurrence and development of colorectal cancer.
To further understand the mechanism of miRNA regulating genes in colorectal cancer development, we performed the gene expression analysis using The Cancer Genome Atlas (TCGA) Colon Cancer cohort.The TCGA gene expression data of colon cancer were derived from the UCSC Xena platform (http://xena.ucsc.edu/).Forty-one normal samples and 453 primary colon tumor samples were included in the gene expression analysis.Cancer development involves various biological processes.Different pathways interact with each other and inf luence the tumor process together.There are complex regulatory relationships between miRNAs and target genes.The experimental results showed that the identified miRNA modules not only have the outstanding ability in disease sample discrimination but also play critical roles in the occurrence and development of colorectal cancer by participating in the perturbations of multiple related pathways.The identified miRNAs are promising as potential

Experiments on other diseases
In addition, we included another three experiments for further verifying the effectiveness of miRMarker.Three challenging tasks were considered: the prognosis of nasopharyngeal carcinoma (NPC), the diagnosis of recurrent implantation failure (RIF) and the severity judgment of COVID-19.Table S7 available online at http:// bib.oxfordjournals.org/gives the details of the involved discovery sets and independent validation sets.
As performed for colorectal cancer, the identified potential biomarkers by miRMarker on the discovery set were validated on the corresponding independent validation set.Table S8 available online at http://bib.oxfordjournals.org/shows the results of the three experiments.For the NPC experiment, miRMarker got an AUC of 0.922 on the discovery set and an AUC of 0.731 on the independent validation set.For the RIF experiment, the AUC values of miRMarker on the discovery set and independent validation set were 0.989 and 0.967, respectively.For the COVID-19 experiment, miRMarker got an AUC of 0.895 on discovery set and an AUC of 1.000 on the independent validation set.
The results of the three experiments further illustrated the effectiveness of miRMarker in identifying the potential biomarkers for different diseases.More details about the three experiments are given in Supplementary Materials available online at http://bib.oxfordjournals.org/.In summary, miRMarker demonstrates its advantages in disease biomarker identification.

CONCLUSION
Identifying the informative miRNAs for the diagnosis and prognosis of diseases, such as cancers, is attractive and challenging.Network-based data analysis methods are important tools in potential biomarker identification.In this study, we proposed a miRNA data analysis method based on multi-view networks and reinforcement learning (miRMarker) to define the potential miRNA biomarkers for diseases.miRMarker constructs the cooperative regulation network and functional similarity networks based on the miRNA expression data and known miRNA-disease relations, respectively.Then, reinforcement learning strategy is employed to define the potential miRNA module biomarkers for discriminating different sample groups.
We compared miRMarker with eight efficient data analysis methods SVM-RFE, WGCNA, DNB, GroupBN, NGTM, QLCD, N-DAG and DDRM on multiple miRNA datasets.miRMarker obtained better performance in disease sample discrimination, indicating that it identified the valuable miRNAs based on the miRNA expression data and knowledge bases.In addition, we applied miRMarker

ξ vu = 1 where
η uv = exp abs sim b iu , b jv b jr ∈B j exp abs sim b iu , b jr , q v=1 η uv = 1 ξ vu = exp abs sim b jv , b iu b iz ∈B i exp abs sim b jv , b iz , p u=1 sim b iu , b jv is the cosine similarity value between b iu and b jv .η uv and ξ vu are the row-normalized and column-normalized weighting factors, respectively.A large sim b iu , b jv leads to a large weighting factor η uv .abs () indicates the absolute value function.

Figure 3 .
Figure 3. Identified modules on colorectal cancer dataset and their performance in disease sample discrimination.(A-E) Identified modules (Modules 1-5).The thickness of the edge indicates the edge weights.(F) Average AUC values of Modules 1-5 in the 5-fold cross validation on training samples.

Figure 4 .
Figure 4. ROC curves of miRMarker on colorectal cancer datasets.(A) ROC cure on the discovery set (GSE108153).(B-D) ROC cures on the independent validations sets from GSE112264, GSE113486 and GSE211692, respectively.

Figure 5 .
Figure 5. Module-disease network.Node sizes represent the node degree.Arrow direction in disease-disease edges indicates that the source node includes the target node.

Figure 6 .
Figure 6.Top 30 enriched KEGG pathways of target genes of hub miRNAs.

Table 1 : Details of miRNA expression datasets Datasets Involved diseases Number of features Number of samples Number of classes
Wallis Test for multi-class samples).If the expression values of f i show the significant difference between different sample groups, it means f i has a high individual distinguishing ability.I t f i = ClosenessCentrality f i is the normalized network topological importance score of f i , defined as the closeness centrality of f i in G FN−MST−kNN .Closeness centrality measures the distances from one node to all other nodes in the network.If the closeness centrality of f i is large, it means that f i is in the central position of G FN−MST−kNN , the network topological importance score of f i is high.Both I e f i and I t f i are normalized using min-max normalization.The comprehensive importance scores of nodes consider both the individual distinguishing ability and network topological property, evaluating the node importance synthetically.The most important g nodes are selected as the initial state nodes.For an initial state node f t(1 ≤ t ≤ m), miRMarker defines a Qtable Q t with all Q values of zero and a set of visited nodes S t only containing f t .The rows and columns of Q t represent the state nodes and action nodes, respectively.Q t f s value i is the normalized expression difference score of f i , representing the individual distinguishing ability in disease sample discrimination.p-value i is the P-value of f i in the Mann-Whitney U Test (Kruskal- AUC (S t ) is the AUC value of S t in ROC analysis.SF (S t ) = 1 + sqrt (1/ |S t |) indicates the size factor of S t .sqrt () is the square root function.|S t | indicates the size of set S t .The reward value R (S t ) is large if S t has a high classification performance and a small size.Q t f s , f a is updated as follows: Number of episodes K, Initial state node f t , Learning rate α, Size limit L.

Table 2 :
Comparison on the miRNA datasets in AUC